Skip to content

update helm chart to use preferred backends for gke#1681

Open
rlakhtakia wants to merge 1 commit into
llm-d:mainfrom
rlakhtakia:feature/preferred-backends-ha
Open

update helm chart to use preferred backends for gke#1681
rlakhtakia wants to merge 1 commit into
llm-d:mainfrom
rlakhtakia:feature/preferred-backends-ha

Conversation

@rlakhtakia

@rlakhtakia rlakhtakia commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What type of PR is this?
/kind feature

What this PR does / why we need it:
Enable helm chart to deploy multiple EPP services and GCPBackendPolicies for gke preferred backends.

Link to external guide

The helm chart deploys 2 EPP deployments, where the "standby/default" pods are attached to the inferencepool. The preferred pods (Active) are deployed after where the user will have to run gcloud commands to attach the backend to the inferencepool. This ensures that when active goes down, passive is always ready to ingest traffic.

Preferred Backends also allows for safe failover by setting traffic capacity to avoid overloading the pod. (Ex. We can set the traffic threshold to be 95% capacity before standby pods get traffic).

Test Results

Active-Passive HA (preferredReplicas: 1, defaultReplicas: 1)

Test Scenario Traffic Profile Succeeded / Total Median Latency (p50) Notes
Baseline Routing 20 sequential requests 20 / 20 (0 dropped) 0.092 s Traffic routes to active Preferred pod ordinal (optimized-baseline-epp-0).
Live Failover / Draining Continuous stream while draining primary tier (capacityScalerPercent: 0) 24 / 25 (1 timed out) 0.092 s 1 timeout (HTTP 504) during ~3s window while health probers mark primary NEG capacity zero; 100% instant failover to warm standby backup pod (optimized-baseline-epp-1).
Graceful Failback Continuous stream while restoring primary capacity (capacityScalerPercent: 100) 25 / 25 (0 dropped) 0.092 s 0 dropped connections. Traffic continues hitting standby pod until GCP probers mark recovered primary pod healthy.
Load Spillover 100 concurrent requests (~40 req/s vs maxRatePerEndpoint: 5) 99 / 100 (1 dropped) 0.100 s Excess burst traffic spills over natively in Envoy memory to standby default tier without dropped connections.

Active-Active HA (preferredReplicas: 2, defaultReplicas: 2)

Test Scenario Traffic Profile Succeeded / Total Median Latency (p50) Notes
Active-Active Baseline 50 concurrent requests across 10 threads 50 / 50 (0 dropped) 0.104 s Traffic actively balances across both active leader pod ordinals (optimized-baseline-epp-0 and optimized-baseline-epp-1).
Scenario A: 1 Active Pod Down Stream during primary pod crash / drain (capacityScalerPercent: 50) 25 / 25 (0 dropped) 0.104 s When 1 active leader pod crashes or drains, the remaining active leader pod handles 100% of Preferred traffic without dropped connections.
Scenario B: Both Actives Down Stream during full primary drain (capacityScalerPercent: 0) 25 / 25 (0 dropped) 0.100 s Once all active Preferred leader pods terminate or drain, 100% of incoming traffic shifts seamlessly to warm standby backup pods (optimized-baseline-epp-2 and optimized-baseline-epp-3).

Optimized Baseline Benchmark

Workload Profile Mean Prompt / Output Success Rate Achieved QPS Total Throughput Output Throughput TTFT (Median) TPOT (Median) Req Latency (Median / p99)
guide_optimized-baseline_1.yaml 6,000 / 1,000 tokens 100% (250 / 250) 36.38 req/s 18,885 tokens/s 13,163 tokens/s 0.156 s 0.047 s 47.16 s / 52.40 s

Release note (write NONE if no user-facing change):

Enable helm chart to deploy epp with GKE preferred backends

@rlakhtakia
rlakhtakia requested a review from a team as a code owner June 17, 2026 20:54
@rlakhtakia
rlakhtakia requested review from liu-cong and vMaroon June 17, 2026 20:54
@github-actions github-actions Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/feature Categorizes issue or PR as related to a new feature. labels Jun 17, 2026
@rlakhtakia
rlakhtakia force-pushed the feature/preferred-backends-ha branch from 2acf74b to 9c46c3d Compare June 18, 2026 00:04
@github-actions github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Jun 18, 2026
liu-cong
liu-cong reviewed Jun 22, 2026
View reviewed changes
Comment thread config/charts/routerlib/values.yaml Outdated
Comment thread config/charts/llm-d-router-gateway/templates/gke.yaml
@rlakhtakia
rlakhtakia force-pushed the feature/preferred-backends-ha branch from 9c46c3d to 763a600 Compare June 22, 2026 22:57
@github-actions github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Jun 23, 2026
@rlakhtakia
rlakhtakia force-pushed the feature/preferred-backends-ha branch from 763a600 to e7d83f0 Compare June 23, 2026 08:21
@github-actions github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Jun 23, 2026
@rlakhtakia
rlakhtakia force-pushed the feature/preferred-backends-ha branch from e7d83f0 to 8bfa141 Compare June 23, 2026 16:42
@github-actions github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Jun 24, 2026
@rlakhtakia
rlakhtakia force-pushed the feature/preferred-backends-ha branch from 8bfa141 to 08fd8ca Compare June 24, 2026 20:22
@github-actions github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Jun 26, 2026
@rlakhtakia

rlakhtakia commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

@ahg-g @liu-cong

Updated this to use stateful sets again, latest tests cover failover and benchmark against optimized baseline.
Also added a link to the external guide on how to use the feature.

@rlakhtakia
update helm chart to use preferred backends for gke
9263898 
Signed-off-by: Radhika Lakhtakia <rlakhtakia@google.com>
@rlakhtakia
rlakhtakia force-pushed the feature/preferred-backends-ha branch from 08fd8ca to 9263898 Compare June 26, 2026 18:09
@github-actions

github-actions Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liu-cong liu-cong liu-cong left review comments
@vMaroon vMaroon Awaiting requested review from vMaroon vMaroon is a code owner automatically assigned from llm-d/router-maintainers

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

kind/feature Categorizes issue or PR as related to a new feature. lifecycle/stale size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rlakhtakia @liu-cong